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The structure and stability of vortices in hybrid atomic-molecular Bose-Einstein condensates is 
analyzed in the framework of a two-component Gross-Pitaevskii-type model that describes the 
stimulated Raman-induced photoassociation process. New types of topological vortex states are 
predicted to exist in the coherently coupled two-component condensates even without a trap, and 
their nontrivial dynamics in the presence of losses is demonstrated. 
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I. INTRODUCTION 

Recent experimental observation of the photoassoci- 
ation of Bose condensed atoms via the stimulated Ra- 
man process and the production of ultracold molecules 
HJ is, undoubtedly, the first step towards the creation of 
a coherently coupled atomic-molecular condensate. Such 
a system would represent a rich ground for exploring a 
variety of the nonlinear atom-optics effects, analogous 
to those known for the processes of parametric up and 
down conversion of light waves H|| . From this point of 
view, the excited topological states of the macroscopic 
wavefunction- vortices— in the atomic-molecular Bose- 
Einstein condensates are of great interest as a direct 
counterpart of the parametric vortices in nonlinear op- 
tics ||,[5|. On the other hand, the study of such topolog- 
ical states is important for understanding the properties 
of composite superfluids, such as the recently proposed 
molecular Bose-Einstein condensate (BEC) coupled to 
a cloud of Fermi atoms H, or the composite atomic- 
molecular BEC (AMBEC) 0. 

From the experimental point of view, it is important 
to not only establish the conditions for the formation of 
topological states, but also identify the states which are 
dynamically stable, i.e. are not destroyed by exponen- 
tially growing collective excitations of the condensate. 
One of the proposed methods for the observation of vor- 
tices involves imaging of the expanding condensate re- 
leased from a trap, with a vortex core growing during 
the expansion. The atomic-molecular condensate rep- 
resents an interesting experimental challenge as the ex- 
cited states may be formed in a dynamically stable, self- 
confined droplet JtJ, after the hybrid condensate is re- 
leased from a trap. 

Assuming further experimental progress towards the 
creation of the molecular and hybrid atomic-molecular 
BECs, in this paper we analyse the dynamics of the para- 
metrically coupled atomic and molecular condensates, in 
the framework of a two-component mean-field model ||] 
that takes into account all types of the mean-field atomic 
and molecular interactions 0. For the first time to our 
knowledge, we study the structure, stability, and dynam- 



ics of topological states in AMBEC, and, in particular, we 
reveal that the dynamically stable two- component vortex 
solitons can be formed in both trapped and untrapped 
AMBEC clouds. Unlike the conventional vortices pre- 
dicted and already observed in atomic BEC, vortex soli- 
tons can exist as stable self-trapped states even without a 
trapping potential, supported by an effectively attractive 
interaction between the parametrically coupled atomic 
and molecular components. 

Another crucial problem is the dynamics of AMBEC 
and its topological states in the presence of dissipation, 
i.e. loss of atoms and molecules from the correspond- 
ing fractions of the hybrid BEC. The formation of BEC 
in the presence of losses due to either a coherent Raman 
photoassociation process or Feshbach resonance has been 
analyzed previously p|,|lO|], however the dissipative dy- 
namics of vortices, earlier studied for a single-component 
atomic BEC [[ll],[L2| , has never been discussed in the con- 
text of hybrid condensates. 

Since, so far, no conclusive experimental data are avail- 
able that would allow us to estimate the rates of the 
condensate losses rigorously, in this work we take them 
into account by including the phenomenological dissipa- 
tive terms, and then considering the dissipation rates for 
which the topological states can still be formed. We ana- 
lyze the dynamics of the vortex solitons in the presence of 
dissipation and reveal that they can exhibit a nontrivial 
decay scenario, with the decay rates much smaller than 
the characteristic loss rates. Since the physical principles 
underlying the existence and stability of these topolog- 
ical states are rather generic, we expect that the simi- 
lar structures can be found in more realistic models of 
the atomic-molecular BEC dynamics, and eventually ob- 
served in experiment. 



II. MODEL 

We study the dynamics of the hybrid atom-molecular 
condenstates, produced by a coherent stimulated Raman 
photoassociation process in a strongly anisotropic trap 
with a tight confinement along one spatial dimension. 
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We model the parametrically interacting condensates by 
a system of two coupled Gross-Pitaevskii (GP) equations 
for the macroscopic wave functions of the atomic (f„) 
and molecular (ty m ) components (see details and refer- 
ences, e.g., in Ref. 0): 
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where the wave functions, time, and spatial coordinates 
are measured in the units of x 3 ^ 2 , wj 1 , and xq = 
(H/mLd^) 1 / 2 , respectively. Here w_l is the characteris- 
tic frequency in the plane (x, y) of the weak confinement 
of the trap for the atomic BEC fraction. Consequently, 
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+ y 2 , and A 2 = u> z /u>j_ 3> 1. In these nota- 
tions, the interaction strengths Uu are measured in the 



units of (hui±) 



and the Raman-induced coupling, 



X, in the units of (Hlu±)~ 1 x 3 ^ 2 . The terms R a ,m on 
the right hand side are the phenomenological loss terms 
which will be discussed and specified below in Sec. 4.2. 

Due to significantly dissimilar scales of spatial con- 
finement in the different directions, and the quasi-two- 
dimensonal geometry of the trap, the form of the conden- 
sate wavefunction in the z direction is completely deter- 
mined by the structure of the harmonic trap. We there- 
fore assume that the wavefunction can be factorized in 
the following way: ^ a , m = ^ a ,m(x,y,t)^ a , m (z), where 
&a,m{z) satisfy the one-dimentional harmonic oscillator 
equations: 
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where v a = (1/2) A 2 z 2 and v m — 2X 2 z 2 . The ground 
state solutions of these equations correspond to e a = A/2 
and e m — A, so that <& a = C a exp(— Xz/2) and <I> m = 
C m exp(— Xz). The normalization conditions J $ 2 m = 1 
yield C a = (A/tt) 1 / 4 and C m = (2A/tt) 1 /4. 

Substituting the factorized wavefunctions into the 
model equations, multiplying the first and second equa- 
tions by (fr a and $ m , respectively, integrating to elimi- 
nate the z dependence, and introducing ipa,m( x i y, t) — 
4>a,m{ x i Vi t) ex P(~ iXt/2), we obtain the final evolution 
equations for the wavefunctions of a "pancake" -shape 
AMBEC: 
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where A ± = d 2 /dx 2 + d 2 /dy 2 , V a (r) = (l/2)r 2 , and 
V m (r) = 1 " 



Here we introduce the renormalized in- 
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teraction strengths: U aa = U aa (X/2ir) 1 / 2 , U mm = 
fWAA) 1 / 2 , U am = C/ Qm (2A/3vr) 1 /2 j x = x (A/2 ? ) 1 /4 
and, correspondingly, the renormalized loss terms R a , m - 
For the sake of clarity, we omit the tilde from the equa- 
tions throughout the following text. One can see that the 
renormalization factors absorbed by the interaction coef- 
ficients as a result of the reduction to the 2D geometry 
are, in fact, close to 1 for typical A ~ 10. 

For definiteness, we consider the case of S7 Rb, for 
which the production of ultracold molecules were re- 
cently demonstrated experimentally jlj , and assume that 
the Raman detuning parameter vanishes, 5 = 0. Us- 
ing the recently reported data for the intra- and inter- 
species scattering lengths 0, we take U aa = 0.062; 
U am = -0.084(±0.07), and x = 1-09. The large un- 
certainty in the value of U am corresponds to the s-wave 
scattering length, a am = — 180ao ± 150ao, recently mea- 
sured experimentally 0]. Since the molecular BEC has 
not been experimentally realized yet, the exact value of 
U m m is not known, although it can be estimated to be of 
the order of the other interaction strengths (see details 
in Ref. Jm|) and, therefore, it cannot be ignored in the 
model (0). 

It has also been previously established that the quali- 
tative behaviour of the system depends on the sign of the 
determinant AU = U aa U m m — U 2 m @,||, which defines 
the relative strength of the cubic interactions. Parameter 
Umm can therefore be used to investigate the system in 
two distinct regimes of the net attractive AU > and 
net repulsive AU < cubic interactions j7j. At the low 
particle numbers, i.e. for the low densities of the conden- 
sate wavefunctions, and in the parameter region where 
AU w 0, the interactions are dominated by the paramet- 
ric, i.e. quadratic, coupling (~ x) which can be effectively 
attractive or repulsive, depending on the relative phase of 
the molecular and atomic components ffl] . The structure 
and dynamical properties of all topological structures in 
the AMBEC is controlled by the complex interplay of the 
quadratic and cubic inter- and intra-component mean- 
field interactions. 



III. VORTEX SOLITONS 

Spatially localized ground-state solutions of the model 
([|) at R a = R m = 0, with and without the trapping 
potential Vj(r), have been analyzed earlier M. Here, we 
are interested in the excited states of the model (||) that 
correspond to the topological vortex states of the hybrid 
two-component condensate. In order to find the vortex 
states, we consider the model equations (j^) without the 
dissipative terms (i.e. R a = R m = 0), and look for the 
radially symmetric solutions with a nontrivial phase in 
the form: 
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^ m (r,6,t)=cf> m (r)e^ 9 ~^ t , (4) 

where (r, 8) are the polar coordinates in the plane of 
the trap, and /J, a ,m are the chemical potentials of the 
corresponding AMBEC fractions. Due to the paramet- 
ric coupling between the atomic and molecular matter 
waves, the existence of stationary states of the AMBEC 
requires the phase-matching conditions, /j, m — 2/i a = 2/i, 
and k m = 2k a = 2n, so that the integer n defines the 
topological charge of the AMBEC vortex soliton via its 
atomic component. 

The spatially localized, radially symmetric solutions of 
the model (||) in the form (|J) can be found by a standard 
numerical relaxation technique. Different types of such 
topological states are summarized in Fig. [j] for the case 
n = 1 (single-charged vortices) . The upper solid curve in 
the middle part of Fig. [l] [that includes the point (d)] cor- 
responds to stable vortices in a trapping potential; such 
vortices are supported by the effectively repulsive cubic 
interaction since AU < 0. These topological states are 
somewhat similar to the vortices of a single-component 
atomic BEC, and they expand when released from the 
trap. At the low particle numbers [see the inset in Fig. 
||, dashed curve with the point (c)j, these vortex solu- 
tions become unstable due to the competition between 
the effectively attractive quadratic and repulsive cubic in- 
teraction. Thus, the repulsive nature of the inter-species 
cubic interaction manifests itself only for large densities 
of the components <f> a and <j> m . 

Thick solid curve in Fig. |l| (and the insert) describes 
a family of the vortices of the single-component purely 
molecular condensate wavefunction ip m [see the case (a)] , 
whereas a dotted curve above it corresponds to an- 
other family of hybrid vortices that bifurcates from the 
purely molecular states. This second family of the two- 
component vortices is supported by an effectively repul- 
sive quadratic nonlinearity allowing such states to exist 
for low particle numbers but only in the presence of a 
trapping potential [cf. an example in (b)]. 

An important feature of the model (||) is the exis- 
tence of untrapped self-localized vortex solitons for fi < 0. 
These structures are supported by an effectively attrac- 
tive quadratic interaction between the atomic and molec- 
ular components. As the parameter fj, decreases, the so- 
lution becomes broader due to the presence of effectively 
repulsive cubic nonlinearity, with a saturation amplitude 
of the wavefunction corresponding to the amplitude of 
the spatially homogeneous (free-space) condensate wave- 
function [see the cases (e) and (f) in Fig. EJ. 
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FIG. 1. Middle: Different families of the stationary one- 

and two-component vortex states in the model (^|). Thin 
solid/dashed - stable/unstable hybrid vortices in a trap, 
dotted - unstable vortices in a trap, dashed - untrapped 
self-localized vortex solitons. Thick solid - vortices of a molec- 
ular condensate. Top and bottom: Examples of the stationary 
profiles of the vortices corresponding to the marked points. 
Dashed-dotted curves in (a)-(d) show the trapping potential. 
Insert: Close-up of the bifurcation region. 

In this paper, we determine the vortex stability by a 
direct numerical propagation, but a more rigorous sta- 
bility analysis will be presented elsewhere. We find that 
the broad vortex solitons [shown, e.g., in Fig. |l](f)] are 
stable, whereas narrow vortex solitons [e.g. those shown 
in Fig. 0(e)] are unstable, in agreement with the earlier 
analysis of the parametric vortices in optics ||. Inter- 
estingly, for the unstable vortices we observe two types 
of the instability-induced dynamics, i.e. the break-up 
into two or three filaments, for larger and smaller parti- 
cle numbers, respectively. This dynamical behaviour is a 
clear signature of the transition between the regime of ef- 
fectively cubic interaction in which the vortex soliton de- 
cays into 2\n\ filaments and the regime of effectively 
quadratic interaction when the vortex soliton breaks up 
into 2\n\ + 1 filaments, as usually observed for quadratic 
parametric beams ]l4| . 
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FIG. 2. Lossless dynamics of the self-trapped vortex soli- 
ton. Top: stable propagation of a broad vortex, the case (f) 
in Fig. Ill (/j. = —1.8). Middle: the symmetry-breaking insta- 
bility induced by the effective cubic nonlinearity, [i — —1.5, 
the case (e) in Fig. 1. Bottom: the symmetry-breaking insta- 
bility induced by parametric interaction, /i = —0.5. Only the 
atomic component is shown. 



IV. DISSIPATION-INDUCED VORTEX 
DYNAMICS 

A. Physical estimates of the losses 

In the formation of ultracold molecules from an atomic 
BEC, the processes of the incoherent production of 
molecules, decays of the atomic excited state, and inelas- 
tic collisions between atoms and molecules, all contribute 
to the the effective loss of particles from the correspond- 
ing fractions of the BEC, resulting in the dissipative dy- 
namics of the localized AMBEC states. 

To take into account all possible losses in the hybrid 
condensate, we introduce the following linear and non- 
linear (i.e., density-dependent) dissipative terms into the 
GP model (|), 

Ra = -Vai>a ~ la\^a\ 2 1pa ~ Jam\lpm\ 2 1pa, 

(5) 

Rm = -V m 1pm ~ lm\^m\ 2 i> m ~ 1am\^a\ 1pm> 

where the dimensionless coefficients, Vj and 7^, stand for 
the decay rates of the atomic excited state (~ v a ) and the 
excited molecules into high bound states of the atomic 
BEC or "hot" (non-BEC) atoms (~ i/ m ), the rates of in- 
elastic collisions and decay of the atomic states (~ j a ), 
and inelastic molecule-molecule collisions and dissocia- 
tion into high bound state or continuum state of the 
molecular BEC (~ 7 m ), and atom- molecule inelastic col- 
lisions resulting in the production of "hot" atoms and 
molecules (~ y am ). 

The linear loss terms in Eq. (^]) are similar to those 
discussed earlier in Ref. ||, however the latter work did 
not take into account the losses due to the rotationally or 



vibrationally inelastic atom-molecule collisions, as well as 
the molecule-molecule collision losses. System (||) with 
the loss terms (||) is formally equivalent to the mathe- 
matical description of the formation of AMBEC through 
Feshbach resonances 0, except for the terms respon- 
sible for the linear losses from the excited atomic state 
(~ vi) which are specific for the AMBEC production via 
the coherent Raman photoassociation process. We stress 
that no conclusive experimental data for the losses that 
would enable the coherent AMBEC production are avail- 
able at the moment, so some reasonable estimates need 
to be made. 

In Eqs. (||), the coefficients for the collisional losses, 7^, 
are measured the units of T — XqU± , and for the typical 
values of x ~ 5 fim, and uj± ~ 10 2 Hz, V ~ 1CP 8 cm 3 /s. 
In what follows, we investigate the effect of relatively 
"weak", 7j = 1CP 3 and "strong", jj = 1CP 1 losses, which 
nevertheless corresponds to physically "strong" inelastic 
collisions rates of T — 10~ n — 10 -9 cm 3 /s. For simplicity, 
we assume that all the inelastic loss rates in the model 
are equal to each other. 

As was estimated in Ref. ||, the linear loss rates that 
still enable the coherent dynamics of the AMBEC are of 
the order of 10 2 s _1 , which corresponds to our dimension- 
less values of Vj ~ 1. Assuming this to be the case, we 
note that the term responsible for the "weakest" nonlin- 
ear losses in our model still an order of magnitude larger 
than the corresponding linear loss, i.e. Ynj ~ 10 3 s" 1 
for the typical condensate density of nj ~ 10 14 cm~ 3 . 
This means that, without loss of generality we can as- 
sume that 7j = Vj in our model, and bear in mind that 
the effect of the linear losses is only significant when the 
particle numbers (and condensate densities) are small. 

The other (strong) loss mechanism for the molecular 
fraction, which is absent in our effectively two-mode con- 
sideration, is the coupling of a molecule to the other, 
excited states of the atomic BEC due to the closeness of 
the excited molecular level to the BEC atom-pair level. 
This loss channel has been recently shown to be responsi- 
ble for the enhancement of the condensate loss rates |L5| . 
One can argue, however, that on the time scales large 
compared with the corresponding rates of decay into the 
higher-order modes, the lowest-order nonlinear excited 
modes of the hybrid AMBEC that we consider here be- 
come decoupled from the rest of the bound state spec- 
trum, and all losses associated with such coupling can be 
effectively incorporated into the phenomenological terms 
proportional to u m . 

B. Lossy dynamics of the vortex soliton 

In the presence of losses, the total particle number in 
both components of AMBEC decreases. First of all, when 
the molecular component is small, the effective equations 
can be decoupled and the dissipative dynamics of two 
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components can be described separately. In particular, 
neglecting the molecular component, and averaging over 
the spatial structure of the atomic BEC wavefunction, 
from Eqs. (^),(||) we obtain the simple equation for the 
density n a — \ip a \ 2 of the atomic component 

which has the analytic solution, 

v a e~ 2Vat 
(C-7 a e lv ^) 

where C — 7a + ^ Q /n a (0). When the effect of linear losses 
can be neglected, the decay of the atomic component 
follows a power law. 

We have conducted the numerical simulations of the 
dissipation-induced dynamics of the vortex soliton vary- 
ing the loss terms. In Fig. |[ we show the decay 
of the maximum amplitude of the atomic component 
n™ = iV'almax °f the two-component vortex soliton in 
the presence of the losses of a different strength. For 
"strong" and "medim" losses (7 = v = 0.1 and 0.01 
respectively), the decay follows approximately the ana- 
lytical dependence (0) and, therefore, the modal (spatial) 
structure of the vortex state is not very important. 
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FIG. 3. Decay of the atomic component amplitude for dif- 
ferent values of losses. The initial state is a self-trapped vortex 
soliton as in Fig. 1(f) and Fig. 2(top). Losses are: 0.001 (dot- 
ted), 0.01 (dashed), and 0.1 (dash-dotted). Thin solid lines 
show the analytic dependence ((?]). 

For weaker losses, we reveal novel features in the vor- 
tex dynamics associated with the interplay between the 
instability-induced vortex fragmentation and the decay 
of the vortex amplitude due to losses. As an example, in 
Fig. H(top) we show the propagation of the initially sta- 
ble, untrapped, vortex soliton [shown as (f) in Fig. 1] in 
the presence of weak dissipation, 7, = z/j = 0.001. While 
this vortex is absolutely stable in the lossless regime (see 
Fig. 2, top), its number of particles decreases and the cor- 
responding vortex solution moves along the dashed curve 
in Fig. 1 from the point (f) to the point (e), entering the 
lower part of the branch corresponding to unstable vor- 
tices. Then, the vortex fragmentation into two or three 



filaments occurs, as shown in Fig. |l|(top). This leads 
us to conclude that the fragmentation to components oc- 
curs when the particle number has dropped far enough to 
place the vortex in the region of instability for stationary 
solutions and, overall, the vortex parameters adiabati- 
cally follow the family of untrapped localized solutions. 

t = t = 40 t = 70 t = 80 t = 85 

t = t = 5 t = 7.5 t = 10 1=12.5 

□BBDD 

FIG. 4. Dynamics of the (initially) stable vortex soliton in 
the presence of losses: (a) weak, ji — Vi = 0.001, and (b) 
strong, 7i = v t = 0.01. 

For comparison, in Fig. |(bottom) we show the vortex 
dynamics for stronger losses, when no fragmentation is 
observed at all. In this case, the vortex soliton does not 
follow a family of localized solutions adiabatically. More- 
over, it appears that the dissipation leads to a switching 
from the fundamental vortex state to one of the higher- 
order vortex modes. The additional "rings" in the den- 
sity distribution observed in Fig. |] (bottom) therefore 
seem to be a manifestation of such a higher-order mode. 

C. Evolution of a vortex released from a trap 

Although different possibilities to create the topolog- 
ical states in experiment may exist, it is probably safe 
to assume that the presence of the confining trap is 
paramount. The question therefore is whether a dynam- 
ically stable topological state of AMBEC created in a 
trap can be transformed into a dynamically stable vor- 
tex in a self-confined AMBEC droplet, after the hybrid 
condensate is released from the trap. We investigate such 
a possiblity by conducting additional numerical studies. 

A trapped vortex, corresponding to the stable family 
shown in the main part of Fig. 1 [the branch that includes 
the point (d)] is released from the trap at t = 0. Wc 
find that for weak or no dissipation this vortex structure 
diffracts rapidly, as shown in Fig. |^(top). However, when 
the losses become large enough, the released vortex state 
gets self-trapped, as shown in Fig. |](bottom). 

In order to verify the effect of self-trapping and get 
a deeper insight into our results, we use the parametric 
plot of the system Hamiltonian H(t) vs. the total particle 
number N(t), which both evolve in time in the presence 
of losses. For weak losses, we reveal that the vortex dy- 
namics follows the "trapped" family of the vortex states 
(dashed line in Fig. ||). The family of the trapped vor- 
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tex states in this plot corresponds to the initial conditions 
of a trapped vortex stationary solution with the trapping 
potential turned off. This vortex state diffracts when it is 
released from the trap due to the dominant repulsive cu- 
bic nonlinearity. However, for stronger losses we observe 
a novel effect when the vortex, being initially released 
from the trap, does not rapidly diffract but, instead, it 
switches to a self-trapped vortex soliton, as shown by a 
dotted curve in Fig. ||. 
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FIG. 5. Dynamics of a vortex state formed in a trap and 
released from the trap at t = 0, under the influence of (a) 
weak (7 = 1/ = 0.001) and (b) strong (7 = 1/ = 0.01) losses. 
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FIG. 6. Parametric plot H(t) vs. N(t) for the vortex evo- 
lution shown in Fig. 5|. Solid (thick and thin) lines - the 
families of the vortex states with and without the trap, re- 
spectively. Dashed and dotted curves show the dynamics of 
the stable vortex initially formed in a trap and released from 
it in the presence of losses, 7 = v = 0.001 and 7 = v — 0.01, 
respectively. 



V. CONCLUSIONS 

In the framework of a two-component mean-field model 
of hybrid coherently coupled condensates, we have ana- 
lyzed the structure, stability, and dynamics of the topo- 
logical states in the atomic-molecular BEC created via 



the induced two-color Raman photoassociation process. 
We have predicted the existence of novel two-component 
vortex solitons supported by the coherent parametric 
coupling between the atomic and molecular BEC frac- 
tions, and demonstrated that such topological states can 
exist even in the absence of a trapping potential. 

Since the losses are known to be very important in the 
formation of the hybrid atomic-molecular condensates, 
we have analyzed the effect of weak and strong losses of 
a different physical origin on the dynamics of the vortex 
solitons, the type of problems never addressed in other 
fields such as nonlinear optics. In particular, for the first 
time to our knowledge, we have described an interplay 
between the nonlinearity-induced vortex fragmentation 
and dissipation-induced vortex decay. Moreover, we have 
revealed that the losses may be very useful to identify 
the self-trapped vortex states in the hybrid condensates 
in experiment, after releasing the atomic-molecular con- 
densate from a trap. 

Although our analysis is valid for a quasi-two- 
dimensional model, the generalization to a spherically 
symmetric case seems straightforward. Additionally, in 
spite of the limited validity of the two-component model, 
we believe that the similar types of topological states can 
be found in more realistic models of hybrid condensates, 
as well as in other types of coherently coupled mixtures 
of the multi-component condensates such as spinor Bose- 
Einstein condensates. 

The work has been supported by the Planning and 
Performance Fund of the Institute of Advanced Studies 
at the Australian National University. 
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